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Throughout  the  acquisition  process  there  are  numerous  opportunities  to  drop  the 
ball.  Using  sound  business  and  systems  engineering  practices  ensures  essential 
information  is  passed  to  enable  better  performance  and  lower  cost. 
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TECHNOLOGY 
MANAGEMENT  BEST 
PRACTICES: 

REDUCING  TECHNOLOGY 
IDENTIFICATION, 
EVALUATION,  AND 
SELECTION  COSTS 


Stephen  J.  Moretto 


We  live  in  an  uncertain  environment.  Military  requirements  are  constantly  evolving 
to  keep  pace  with  advances  in  technology  and  changes  in  the  forces  that  threaten 
Americans.  Assessing  the  success  and  effectiveness  of  today's  complex  systems 
is  becoming  an  increasingly  challenging  and  costly  problem.  The  services  are 
being  asked  to  increase  performance  and  reduce  lifecycle  costs.  By  beginning 
with  the  end  in  mind  and  using  existing  tools  as  a  starting  point,  stakeholders 
can  contribute  to  reduced  program  support  costs,  better  justification  of  source 
selections,  and  better  end  products.  Examples  will  demonstrate  how  alternatives 
can  be  prioritized  to  fit  within  budget  constraints,  how  to  link  capabilities  to  cost, 
and  how  to  efficiently  select  the  best  value  among  competing  complex  systems. 


Designing  and  placing  a  weapons  system  under  contract  is  time-consuming  and 
costly.  It  is  done  best  when  the  diverse  disciplines  work  in  an  integrated  fashion. 
This  article  takes  knowledge  from  the  technical  and  acquisition  communities 
and  identifies  tools,  techniques,  and  processes  than  can  enable  them  to  work  in  a  more 
integrated  fashion.  The  technology  selection  process  occurs  near  the  beginning  of 
the  research  and  development  process  and  is  run  by  the  technical  community,  while 
the  source  selection  process  occurs  much  later  in  the  acquisition  process.  The  two 
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processes  have  some  commonality  in  that  they  use  metrics  to  evaluate  the  value  of 
technologies  and  proposals. 

Time  and  cost  in  a  proposal  evaluation  can  be  reduced  by  improving  the  linkages 
between  technology  evaluation  and  source  selection  processes.  The  cost  of  identifying, 
evaluating,  and  selecting  weapon  system  technologies  is  measured  in  terms  of 
government  labor-hours  expended,  travel  expenses,  over-runs,  delayed  schedules, 
performance  shortfalls,  and  missed  opportunities.  This  article  will  identify  some  of  the 
opportunities  to  improve  organization,  requests  for  proposals,  and  evaluation  processes 
and  criteria  that  are  integral  to  the  procurement  of  complex  weapon  systems. 

The  result  of  improvements  will  focus  our  resources  on  that  which  is  of  greatest  value. 
The  goal  is  to  reduce  both  government  and  contractor  labor-hours  expended  and  improve 
communication  between  contractors  and  government.  Improved  communication  will 
focus  efforts  on  maximizing  performance  within  budget  constraints. 

By  begirming  with  the  end  in  mind,  the  contractor  can  spend  less  time  trying  to  interpret 
requirements,  and  spend  more  time  focusing  on  design  and  delivery  of  products  that 
offer  more  value  for  the  dollar.  Government  costs  shall  decrease  due  to  better-defined 
and  automated  processes.  The  government  and  contractor  would  also  reduce  program 
risks  through  more  clearly  defined  contract  performance  requirements.  The  remainder 
of  the  article  will  describe  the  military  requirement  generation  process,  the  technology 
and  source  selection  process,  and  suggest  improvements  to  these  processes. 


OVERVIEW  OF  REQUIREMENTS  GENERATION 

The  weapons  system  acquisition  process  starts  with  the  need  for  national  and 
economic  security,  which  translates  into  requirements  for  complex  military  systems  with 
the  capabilities  to  accomplish  missions.  As  shown  in  Figure  1 ,  requirements  originating 
at  the  top  level  and  science  &  technology  programs  originating  at  the  bottom  level 
lead  to  the  development  of  integrated  systems  that  accomplish  our  goals  of  national 
and  economic  security.  The  science  and  technology  efforts  form  the  foundation  of  the 
development  process  and  become  capabilities  that  are  combined  into  national  assets 
such  as  ships  and  planes.  Technology  identification,  evaluation  and  selection  go  on  at 
each  level  of  system  development. 

This  article  asserts  that  the  best  results  are  achieved  when  a  System-of-Systems 
approach  is  used.  This  is  where  you  consider  how  all  the  pieces  work  together 
and  optimize  the  system  as  a  whole,  instead  of  just  focusing  on  subsystems.  The 
challenge  is  getting  both  decision  makers  and  subsystem  designers  to  understand  the 
interrelationships  between  subsystems  and  the  effect  of  the  performance  on  the  whole 
system.  For  example,  engine  designers  may  want  a  big  powerful  engine,  but  if  they 
get  the  space  for  the  big  engine,  there  may  not  be  sufficient  space  for  a  gas  tank  large 
enough  to  enable  the  vehicle  to  perform  the  mission.  This  concept  can  be  used  at  each 
level  of  system  design. 
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FIGURE  1.  REQUIREMENTS  GENERATION  AND  FULFILLMENT 
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PROBLEMS  AND  CHALLENGES 

Assessing  today’s  complex  systems  is  becoming  more  challenging.  Demands  and 
constraints  are  increasing,  despite  relief  from  Department  of  Defense  (DoD)  acquisition 
process  requirements.  Figure  2  shows  some  of  the  demands  and  constraints  in  assessing 
today’s  complex  systems.  The  challenge  is  to  link  the  requirements  generation  process 
to  the  generation  of  the  request  for  proposals  (RFPs),  and  to  incorporate  performance- 
based  standards  into  the  RFPs  and  source-selection  criteria. 


SELECTION  PROCESSES 

Selection  processes  vary  during  the  procurement  of  a  weapon  system.  The  DoD  uses 
various  systems  to  decompose  top-level  requirements,  match  them  to  systems,  perform 
tradeoffs  and  select  quantities  of  equipment  to  procure.  The  information  flows  up  from 
the  program  office  designers,  who  use  detailed  models  to  determine  and  select  the  best 
combination  of  capabilities  within  cost  constraints.  The  best  combination  of  capabilities 
is  determined  by  using  a  requirements  generation  process  that  involves  the  technical 
community,  users,  and  the  program  office.  The  technology  identification,  evaluation  and 
selection  process  need  to  be  performed  early  in  the  development  cycle.  This  process 
generates  detailed  results  that  should  be  used  later  on  in  the  procurement. 

Source  selection  is  the  business  side,  a  part  of  the  contracting  process,  in  which  the 
government  contracts  department  leads  an  effort  to  choose  the  best  source  to  produce 
the  weapon  system.  The  contracting  process  starts  with  market  surveillance  to  see  what 
technologies  and  vendors  are  available  on  the  open  market.  This  is  the  best  phase  for 


Assessing  the  success  and  effectiveness  of  today’s  compiex  systems 
becomes  an  increasingiy  chaiienging  and  costiy  probiem. 


DEMANDS 


Increased  Performance 
Lower  System  LCC  Costs 
Longer  Operating  Capacities 
Improved  Productivity/Efficiency 


CONSTRAINTS 


Limited  Resources 
Scant/Unknown  Data 
Identification/Resolution  of  Conflicts 
Resource  Allocation 


Need  for  INTEGRATED  and  SYSTEMATIC  framework  that  can  identify 
potential  problem  areas,  assess  system  effectiveness,  and 
aid  in  resource  allocation  and  decision-making  processes 


FIGURE  2.  DEMANDS  AND  CONSTRAINTS 


20 


TECHNOLOGY  AMNAGEMENT  BEST  PRACTICES 


designers  and  engineers  to  become  involved  in  the  business  side  of  the  contracting 
process  source  selection  because  it  allows  the  technical  and  contracting  communities 
to  use  the  results  of  the  technology  evaluation  process  throughout  the  contracting 
process. 

After  market  surveillance,  the  contracts  department  prepares  an  RFP  that  is  sent 
out  for  contractors  to  bid  on,  and  the  government  then  evaluates  and  selects  the  best 
proposal.  One  challenge  is  to  get  technical  personnel  involved  in  the  front  end  to  assist 
the  contracts  department  in  the  preparation  of  the  RFP  and  determination  of  source 
selection  criteria.  Using  more  detailed  performance-based  criteria  would  result  in  better 
proposals  that  could  be  evaluated  efficiently  (reducing  labor  by  technical  evaluators), 
provide  better  justification  of  source  selection  (reducing  costs  related  to  protests), 
and  better  contractor  performance  downstream  (providing  better  cost  and  schedule 
performance  due  to  better  up-front  understanding  of  requirements). 

By  beginning  with  the  end  in  mind,  DoD  can  reduce  costs  and  end  up  with  more 
effective  systems.  When  technical  experts  are  engaged  later  in  the  process,  technical 
requirements  and  evaluation  factors  are  less  well  defined,  making  it  more  difficult  for 
contractors  to  understand,  and  more  difficult  for  government  personnel  to  evaluate. 
Source  selection  criteria  have  been  limited  in  scope  in  the  past  because  of  the  huge 
effort  required  to  compile  and  integrate  the  results.  Fortunately,  processes  and  electronic 
tools  have  been  developed  to  compile  and  integrate  proposal  evaluation  results.  Using 
these  tools  is  sure  to  reduce  cost  and  increase  effectiveness  of  both  the  source  selection 
and  technology  selection  efforts.  The  remainder  of  the  article  will  outline  the  processes 
and  tools  that  can  be  used  in  any  selection  process. 


REQUIREMENTS  IDENTIFICATION  AND 
DEFINITION  PROCESS  IMPROVEMENT 

Systems  engineering  requirements  analysis  is  a  best  practice  in  the  DoD.  DoDI 
5000.2  states  that  “throughout  the  acquisition  process  the  program  office  shall  work  to 
establish  and  refine  operational  design  requirements  that  result  in  the  proper  balance 
between  performance  and  cost  constraints.”  Requirements  analysis  shall  be  conducted 
iteratively  with  functional  analysis  to  develop  and  refine  performance  requirements  and 
external  interfaces.  This  also  shall  provide  traceability  to  user  and  design  requirements. 
The  Technology  Identification,  Evaluation,  and  Selection  (TIES)  process  in  the  next 
section  satisfies  much  of  this  requirement. 

Requirements  development  begins  with  Warfighter  input  and  ends  with  establishment 
of  formal  requirements.  The  TIES  process  works  best  when  it  solicits  information  from 
all  sources  in  appropriate  forums.  Actively  seeking  participation  by  analysts,  military 
users,  industry,  engineers,  academia,  and  others  in  the  life  cycle  of  a  technology  is  a  best 
practice  that  can  have  high  return  on  investment.  Soliciting  and  getting  participation 
from  a  broad  network  is  the  first  step  in  developing  better  processes.  These  networks 
can  help  in  articulating  and  refining  requirements,  which  in  turn  helps  in  identifying 
technologies  that  maximize  capabilities  and  minimize  cost. 
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A  four  phase  process  for  requirements  generation  that  worked  in  a  major  ship  design 

program  is  described  below: 

1.  Prioritize  attributes:  In  ship  design,  the  Universal  Naval  Task  List  (UNTL)  and 
Universal  Joint  Task  List  (UJTL)  can  be  used  as  a  starting  point.  Each  task  was 
prioritized  by  experts  using  an  analytical  hierarchy  process  and  then  each  technology 
was  rated  as  to  how  it  contributed  to  the  task.  The  life  cycle  cost  of  each  technology 
was  evaluated  as  well. 

2.  Trade  study  plan:  The  plan  outlines  the  design  features  and  sets  of  technologies 
that  are  to  evaluated.  This  is  like  choosing  different  cars  with  numerous  options 
packages  to  evaluate. 

3.  Cost  and  Performance  Data:  Once  trade  studies  are  defined,  cost  and  performance 
data  are  generated  with  models  and  evaluated. 

4.  Requirements  Documents/Specifications:  Requirements  are  defined  in  terms  of 
minimum  and  desired  capabilities.  This  is  one  of  the  key  areas  to  link  to  the  REP 
later  in  the  program.  Care  should  be  made  to  preserve  the  details  that  are  used  to 
generate  the  top-level  capabilities  in  the  requirements  document.  These  details 
should  be  used  in  the  RFPs  and  source  selection  criteria.  An  example:  Top-level 
capabilities  are  documented  and  each  is  described  by  up  to  eight  attributes. 
Attributes  are  defined  and  described  by  up  to  six  quantifiable  metrics.  Thus  a  very 
complex  system  may  have  eight  top-level  capabilities,  up  to  64  attributes,  and  over 
380  metrics.  The  metrics  should  be  included  in  the  REP  and  used  as  a  framework 
for  the  source  selection  criteria  later  on  in  the  program. 


NOTIONAL  REQUIREMENTS  ANALYSIS  PROCESS 

One  Navy  program  office  put  in  place  a  technology  evaluation  methodology 
implementing  the  Quality  Function  Deployment  (QFD)  process  to  prioritize 
technologies.  The  overall  process  related  a  set  of  warfighting  tasks  to  technology  areas 
and  then  evaluated  the  programmatic  aspects  of  each  technology.  To  evaluate  the 
programmatic  aspects  of  the  technology  programs,  a  hierarchical  programmatic 
objective  model  was  established  using  seven  programmatic  top-level  objectives  defined 
by  the  Office  of  the  Chief  of  Naval  Operations  (OPNAV)  sponsor.  The  objectives  are: 
Affordability,  Statutory  and  Regulatory  Guidance,  Force  Structure,  Force  Modernization, 
Risk,  Technology  Transition,  and  Industrial  Base.  Of  these.  Affordability,  defined  in 
this  process  as  Total  Ownership  Cost  (TOC),  was  weighted  the  heaviest  in  the  model. 

Each  technology  program  was  then  evaluated  against  the  programmatic  criteria.  The 
result  was  a  prioritized  list  of  technology  programs,  each  with  the  appropriate  mix 
of  warfighting  and  programmatic  objectives.  Working  with  representatives  from  the 
Navy,  the  team  first  prioritized  a  set  of  warfighting  tasks.  These  tasks  were  defined  by 
the  UJTE,  which  defines  the  strategic,  theater,  and  tactical  tasks  for  naval  forces.  The 
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list  of  tactical-level  tasks  was  tailored  to  reflect  potential  tasks,  then  prioritized  using  a 
pair-wise  comparison  process. 

The  next  step  deflned  a  set  of  performance  attributes,  such  as  speed,  range,  and 
vulnerability.  Each  attribute  was  then  correlated  to  each  of  the  warflghting  tasks  to 
determine  the  relative  importance  of  the  attributes  in  meeting  the  prioritized  tasks. 
Likewise,  a  set  of  technology  areas  was  deflned  and  correlated  to  the  attributes,  one  by 
one.  This  resulted  in  a  set  of  prioritized  technology  areas  which  could  be  traced  back  to 
their  impact  on  warflghting,  through  attributes,  to  tasks.  Each  technology  project  was 
then  mapped  to  its  most  appropriate  technology  area. 

Figure  3  shows  a  methodology  for  linking  technologies  to  requirements  and 
speciflcations  used  in  a  major  ship  design  program.  The  team,  consisting  of  functional 
experts  and  a  group  facilitator,  used  a  QFD  process.  The  approach  used  numerous 
pair-wise  comparisons  that  were  entered  into  software  to  prioritize  technologies.  This 
approach  proved  effective  in  the  design  phase  for  technology  selection  but  was  also 
very  time  consuming  and  expensive  to  implement. 

The  Defense  Systems  Management  College  (DSMC)  teaches  the  technical 
community  in  its  Advanced  Production  and  Quality  Management  Course  (PQM-301) 
that  the  QFD  process  is  a  highly  effective  way  to: 

■  Capture  requirements. 


Evaluation  process  uses  a  Quality  Function  Deployment  (QFD)  approach 


Phase  I 

Warfighting  Tasks  to 
Attributes 


HOWs 

(Voice  of  Designers) 


WHATs 

(Fleet  &  OPNAV) 


Phase  II 

Attributes  to 
Technoiogy  Areas 


Series  of  steps 
in  each  phase 


Phase  I 


Technoiogy  Areas  to 
individuai  Technoiogies 


Phase  IV 

Programmatics  to 
individuai 
Technoiogies 


Prioritized  technology  list 
Investment  strategy 


FIGURE  3.  TECHNOLOGY  EVALUATION  METHODOLOGY 
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■  Structure  or  respond  to  RFPs, 

■  Develop  source  selection  criteria, 

■  Identify  key  areas  for  contractual  awareness  (incentive  or  award  fee), 

■  Structure  acquisition  strategies, 

■  Manage  the  document  and  decision  process,  and 

■  Apply  a  structured  systems  engineering  process. 


Reientfy,  the  Office  of  Naval  Reseanh  Affordability 
Measurement  and  Predktion  Program  completed 
development  of  a  new  generation  of  tools  and  techniques 
that  integrate  established  techniques  from  private  industry 
into  what  they  call  the  Technology  Identification^  Evaluation, 
and  Selection  (TIES)  Process. 


The  technical  community  uses  the  QFD  to  capture  requirements  and  to  structure 
the  systems  engineering  process.  The  DoD  application  of  QFD  to  structure  proposals, 
develop  source  selection  criteria,  or  set  up  contractor  incentives  is  not  wide.  The 
application  of  QFD  by  program  managers  or  contracts  personnel  is  not  widespread  in 
the  acquisition  process  for  several  reasons: 

■  The  length  of  time  between  QFD  application  in  the  technical  community  and 
acquisition-related  applications, 

■  QFD  is  not  generally  taught  outside  the  technical  community, 

■  Details  of  QFD  do  not  reach  the  higher  management  levels, 

■  QFD  can  require  large  amount  of  resources,  which  may  limit  its  application  to  larger 
programs,  and 

■  Pair-wise  comparisons  of  proposals  in  proposal  evaluations  are  limited  by 
regulations. 
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TECHNOLOGY  IDENTIFICATION,  EVALUATION, 

AND  SELECTION  (TIES)  PROCESS 

Recently,  the  Office  of  Naval  Research  Affordability  Measurement  and  Prediction 
Program  completed  development  of  a  new  generation  of  tools  and  techniques  that 
integrate  established  techniques  from  private  industry  into  what  they  call  the  Technology 
Identification,  Evaluation,  and  Selection  (TIES)  Process.  The  TIES  process  develops 
stronger  and  more  integrated  links  between  technology,  requirements,  design,  and 
procurement  processes.  Its  use  in  program  management  will  result  in  a  decrease  in 
the  time  and  cost  to  implement  programs  as  well  as  facilitate  better,  more  capable 
designs. 

Cost-conscious  industry  and  government  personnel  are  very  concerned  with 
investment  cost  and  risk  associated  with  developing  and  infusing  new  technologies. 
Because  of  the  focus  on  the  bottom  line,  many  companies  and  government  development 
efforts  dismiss  new,  innovative,  and  revolutionary  designs  due  to  the  potential  risk 
of  profitability  loss  and  budget  constraints.  A  comprehensive  structured  process, 
applicable  to  any  system,  was  needed  to  show  companies  and  government  designers 
whether  a  technology  can  be  shown  to  improve  a  system  at  low  risk.  The  TIES  method 
was  developed  by  the  Office  of  Naval  Research  (ONR)  and  the  Georgia  Institute  of 
Technology  a  few  years  ago  to  assess  very  aggressive  vehicle  concepts  that  could  not 
meet  cost  and  performance  goals  with  present  day  technologies.  The  TIES  process  is  a 
systematic  approach  to  finding  affordable  technical  solutions  that  quantify  impacts  in 
terms  of  performance,  cost,  and  risk  in  the  early  phases  of  design.  This  method  takes 
existing  techniques  from  the  various  sources  listed  below  and  integrates  them  into  the 
TIES  method. 

ESTABLISHED  TECHNIQUES 

■  Response  Surface  Method  (Biology,  Operations  Research) 

■  Design  of  Experiments  (Agriculture,  Manufacturing) 

■  Quality  Function  Deployment,  Pugh  Diagram  (Automotive) 

■  Morphological  Matrix  (Forecasting) 

■  Multiple  Attribute  Decision  Making  (MADM)  techniques  (U.S  Army,  DoD) 

■  Uncertainty/Risk  Analysis  (Control  Theory,  Finance) 

■  Technology  Readiness  Eevels  (NASA) 
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FIGURE  4.  TECHNOLOGY  IDENTIFICATION,  EVALUATION, 
AND  SELECTION  (TIES)  PROCESS 
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ONR  INNOVATIONS 

■  Technology  Identification,  Evaluation,  Selection  (TIES) 

■  Proposal  Evaluation  Tool  (PET) 

Figure  4  shows  the  general  structure  of  the  TIES  method  and  is  intended  to  be  a 
stand-alone  fiowchart  to  capture  all  aspects  of  the  TIES  method.  The  primary  axis 
shows  the  eight  main  elements  that  compose  TIES.  It  begins  with  problem  definition 
and  moves  through  technology  selection  that  best  meets  customer  requirements.  Above 
the  primary  axis  are  various  techniques  or  inputs  suggested  or  required  to  accomplish 
each  step.  Below  the  axis  are  the  primary  results  of  each  step. 


THE  PARETO  ANALYSIS  OF  COST  DRIVERS 

A  critical  goal  of  many  programs  going  through  the  milestone  process  is  to  balance 
cost  constraints  against  desired  performance  enhancements.  These  traditional  life  cycle 
cost  formats  provide  data  in  a  summary  fashion  but  provide  little  insight  into  variables 
which  drive  costs,  especially  at  the  system-  or  design-feature  level.  Thus,  the  data  were 
useful  to  only  some  of  the  stakeholders. 


4  cr/f/ca/  goal  of  many  programs  going  through  the 
milestone  protess  is  to  balante  rost  ronstraints  against 
desired  performame  enhamements. 


The  solution  is  to  convert  data  to  a  system-level  breakout.  This  is  useful  to  both 
designers  and  decision-makers.  The  new  comprehensive  data  structure  facilitates  a  far 
more  insightful  view  of  the  systems’  life  cycle  cost  drivers.  In  order  to  do  meaningful 
analysis,  traditional  life  cycle  cost  breakdowns  should  be  decomposed  into  their 
subordinating  elements  and  ranked  by  cost  element,  from  highest  to  lowest  cost.  This 
allows  one  to  identify  (using  the  Pareto  technique)  the  hierarchy  of  cost  drivers  at  the 
ship  system  level.  This  information  can  play  an  important  role  in  justifying  a  program 
investment  strategy  and  determining  the  phasing  of  an  acquisition  strategy. 


EVOLUTIONARY  ACQUISITION  STRATEGY 

An  evolutionary  acquisition  strategy  is  an  excellent  technique  for  a  program  to  use 
when  adapting  its  approach  to  a  cost-constrained  environment.  The  life  cycle  cost 
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analysis  serves  its  most  important  function  in  providing  the  information  utilized  in 
developing  evolutionary  strategies.  Initially,  most  programs  start  with  a  clean-sheet 
design  or  a  one-step  approach  to  a  new  design,  but  often  the  up-ffont  cost  precludes 
programs  from  being  funded  at  desired  levels  in  the  near-term. 

As  an  alternative  to  the  clean-sheet  approach,  an  evolutionary  strategy  of  gradual 
change  can  succeed.  A  key  concept  in  this  approach  is  to  invest  in  the  top  cost  and 
performance  drivers  first,  where  the  most  significant  cost  reductions  and  performance 
enhancements  can  be  achieved  at  the  front  end.  Besides  cost  drivers,  programs  also 
must  consider  risk  and  technological  maturity  in  determining  an  evolutionary  approach. 
Thus,  follow-on  improvements  can  be  made  as  risk  reduction  and  technology  maturation 
efforts  become  successful. 


DESIGN  OF  EXPERIMENTS 

In  order  for  the  program  office  to  best  utilize  resources,  it  used  principles  of  the  design 
of  experiments  in  developing  its  alternatives.  The  objective  in  design  of  experiments  is 
to  gain  the  maximum  amount  of  knowledge  with  the  minimum  expenditure  of  effort.  It 
was  developed  in  the  1 920s  and  is  widely  used  in  the  industrial  and  systems  engineering 
field  and  is  one  of  the  most  powerful  tools  integrated  into  the  TIES  method.  It  creates 
product  and  process  designs  that  are  relatively  “resistant”  to  variations  emanating  from 
manufacturing  environment.  Its  key  benefits  include: 

■  Decreasing  total  engineering  cycle-time, 

■  Improving  the  performance  of  key  customer  requirements, 

■  Creating  product/process  designs  which  are  robust  to  any  possible  noise  factors, 
and 

■  Improving  the  manufacturability  of  a  product  design. 

Design  of  experiments  helps  to: 

■  Identify  how  feasible  or  viable  the  baseline  concept  is  with  respect  to  the  constraints 
and  objectives; 

■  Identify  which  constraints  are  hurting  the  baseline  concept  (i.e.,  “show-stoppers”); 

■  Identify  how  much  improvement  is  needed  so  that  proper  technologies  can  be 
selected  for  infusion;  and 

■  Establish  an  optimal  configuration  for  technology  infusion. 
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With  that  in  mind,  a  program  office  should  create  a  design  feature  matrix  that  lays 
out  groups  of  design  features  and  technologies  and  assigns  them  to  a  select  number  of 
alternative  concepts.  The  mix  of  technologies  should  be  chosen  to  cover  the  range  of 
capabilities  and  limit  the  number  of  alternatives  that  need  to  be  studied.  In  this  way, 
a  program  office  can  save  time  and  effort  by  focusing  on  the  most  relevant  concepts. 
The  design  feature  matrix  also  links  the  performance  requirements  to  proposed  design 
features  and  technologies.  The  matrix  should  be  a  focal  point  for  designers,  cost 
engineers,  and  decision  makers  in  the  Analysis  of  Alternatives  (AOA). 


The  Office  of  Naval  Researth  Affordability  Measurement 
and  Prediition  Program  and  Georgia  Institute  of  Tethnology 
have  developed  a  tool  railed  Teihnology  Frontiers  that  ran 
help  determine  the  best  rombination  of  terhnologies  given 

design  ronstraints. 


The  Office  of  Naval  Research  Affordability  Measurement  and  Prediction  Program 
and  the  Georgia  Institute  of  Technology  have  developed  a  tool  called  Technology 
Frontiers  that  can  help  determine  the  best  combination  of  technologies  given  design 
constraints.  It  identifies  an  ideal  solution  established  from  summation  of  “best” 
individual  effectiveness  parameters  and  alternative  compromise  solutions  which  also 
can  be  chosen  depending  on  the  priorities  of  decision-makers.  This  tool  can  improve 
the  design  of  experiments  and  can  reduce  the  effort  in  evaluating  alternatives  within 
program  offices. 


INFLUENCE  AND  IMPACT  ANALYSIS 

Perhaps  the  most  difficult  aspects  to  quantify  in  a  program  are  the  synergistic 
and  negative  effects  technologies  can  have  on  each  other  and  the  total  system.  The 
concept  of  systems  engineering  was  discussed  earlier,  where  the  goal  is  to  optimize  the 
total  system  rather  that  some  of  the  parts.  Complex  systems  can  have  many  different 
alternative  components.  Each  component  can  impact  other  elements  of  the  system. 
One  way  to  get  started  in  understanding  the  interrelationships  between  requirements, 
technologies,  costs,  and  functional  areas  is  to  develop  infiuence  diagrams.  The  goal 
of  infiuence  and  impact  analysis  is  to  provide  an  effective  means  of  specifying  and 
evaluating  new  architectures  for  large  scale  complex  systems.  Integrated  process 
teams  can  fill  out  infiuence  forms  for  processes  or  systems.  Utilizing  these  forms  will 
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FIGURE  5.  INFLUENCE  FORM  EVALUATING  A  SYSTEM 
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help  focus  each  functional  area  on  optimization  of  the  total  system  rather  than  on  its 
individual  parts. 

Figure  5  shows  one  type  of  influence  form.  The  left  side  of  the  charts  shows  the 
capabilities  and  attributes  that  the  requirements  team  generated.  The  top  of  the  form  lets 
you  All  in  what  system  level  components  are  influencing  an  alternative.  The  right  side 
lists  the  cost  categories,  Integrated  Process  Teams,  and  other  physical  characteristics 
that  are  influenced  by  the  system.  As  you  follow  the  lines  inward,  impacts  are  identifled 
as  plus  or  minus.  This  allows  designers  a  first  order  look  at  the  bigger  picture  and  to 
consider  the  influence  of  their  subsystem  on  other  subsystems.  The  big  payoff  here  is 
that  the  process  of  Ailing  out  this  form  can  serve  as  a  catalyst  for  functional  areas  to 
communicate  more  effectively. 


PROPOSAL  EVALUATION  TOOLS 

Proposal  Evaluation  Tools  (PETs)  such  as  Expert  Choice,  Eogical  Decisions  for 
Windows,  and  ONR’s  PET  are  very  effective  in  providing  the  framework  for  efficiently 
evaluating  technology  options  that  often  are  very  complex  with  many  permutations.  The 
PET  is  a  web-enabled  decision  analysis  tool  that  facilitates  evaluations  of  competing 
alternatives.  The  tool  is  able  to  automate  much  of  the  proposal  evaluation  process, 
conform  to  proposal  evaluation  regulations,  and  reduce  proposal  evaluation  costs.  It 
was  funded  by  ONR  and  developed  by  Tecolote  Research  Incorporated.  The  PET  is 
free  to  government  users  because  it  was  developed  using  government  funds. 


PROPOSAL  EVALUATION  TOOL  APPLICATIONS 

The  PET  was  applied  to  the  NAVSEA  Corporate  Productivity  Fund,  other  Navy 
Programs,  and  demonstrated  its  practical  abilities  in  prioritizing  multiple  projects 
for  evaluating  and  funding  technologies.  It  efficiently  allocated  scarce  resources  to 
a  number  of  proposals.  Some  Office  of  Naval  Research  codes  have  used  it  to  help 
prioritize  and  select  research  projects  to  fund  each  year.  The  tool  is  unique  because 
it  allows  the  funding  stream  of  many  alternatives  to  be  inputted  and  considered  in  its 
optimization  sequences.  The  tool  is  suitable  for  formal  proposal  evaluations  because  it 
uses  a  question-based  evaluation  process,  while  also  proving  suitable  for  use  in  AOA. 
The  application  of  the  tools  will  be  discussed  further  in  the  demonstration  of  results 
section. 


BENEFITS  OF  USING  THE  PROPOSAL  EVALUATION  TOOL 

The  tool  is  easy  to  use  and  produces  results.  Setting  it  up  gives  structure  to  proposal 
evaluations,  while  significantly  reducing  the  cost  and  time  it  takes  for  proposal 
evaluations  and  source  selections.  Evaluations  can  be  done  securely  via  the  web  or 
from  a  common  server.  Proposal  evaluator  travel  expenses  are  eliminated  if  evaluations 
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are  done  via  the  secure  web.  Also,  significant  billable  hours  charged  by  evaluators  are 
greatly  reduced  due  to  the  efficient  process  and  structure  that  emerge  from  using  the 
tool.  It  can  be  used  by  any  number  of  evaluators,  which  is  important  in  large  source 
selections.  Evaluations  are  automatically  updated  and  integrated  so  that  leadership  can 
see  results  develop  in  real  time. 

It  also  allows  a  decision  maker  to  weight  the  decision  variables  when  setting  up 
proposal  evaluation  criteria  and  performing  sensitivity  analysis.  Sensitivity  analysis 
can  be  accomplished  using  both  text  and  graphical  displays.  Decision-makers  can  view 
the  evaluation  scores  of  all  evaluators.  All  top  level  scores,  individual  evaluator  scores, 
and  proposal  scores  can  be  viewed.  The  program  also  identifies  and  can  display  outlier 
scores.  This  allows  decision  makers  to  recognize  blind  spots,  bias,  or  other  issues  that 
may  not  be  identified  using  traditional  proposal  evaluation  methods. 


PROPOSAL  EVALUATION  TOOL  METHODOLOGY 

The  PET  uses  an  easy-to-follow  methodology  that  can  be  done  by  individuals  or 
through  an  integrated  teaming  approach.  The  method’s  basic  steps  include: 

1.  Defining  the  criteria  and  weighting  their  importance, 

2.  Determining  the  questions  and  scoring  for  answers, 

3.  Applying  the  questions  to  a  set  of  proposals  to  obtain  relative  values,  and 

4.  Prioritizing  proposals  by  maximizing  relative  values  using  integer  linear 
programming. 


RESULTS  OF  PROPOSAL  EVALUATION  TOOL  APPLICATION 
TO  NAVSEA  PROGRAMS 

With  two  days’  effort,  the  results  of  an  AOA  and  prioritization  of  NAVSEA  corporate 
productivity  fund  projects  were  reproduced  with  technical  assistance  from  Tecolote 
Research.  The  PET  first  allows  the  input  of  pair-wise  comparisons  as  a  starting  point, 
and  then  uses  performance-based  questions  to  develop  relative  rankings  of  alternatives. 
It  is  this  second  step  that  allows  the  analysis  to  transition  from  requirements  evaluation 
to  formal  proposal  evaluations.  The  results  below  show  the  analysis  and  illustrate  the 
process  and  the  outputs  of  the  tool. 


AOA  ANALYSIS  USING  THE  PROPOSAL  EVALUATION  TOOL 

The  AOA  evaluated  70  different  alternatives  with  eight  capabilities,  31  attributes, 
and  177  metrics.  These  parameters  were  entered  into  the  proposal  evaluation,  and 
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pair-wise  comparisons  were  made  to  determine  the  relative  importance  of  each 
parameter.  The  metrics  were  converted  to  performance-based  questions  whose  answers 
could  be  scored. 

Once  all  the  criteria  are  evaluated  for  each  proposal,  the  tool  generates  scores  for  each 
attribute.  A  color  code  is  assigned  to  each  attribute  score  based  on  user  specifications. 
The  tool  then  takes  the  raw  scores  and  calculates  the  relative  value  of  each  alternative 
and  displays  them,  enabling  one  to  select  the  one  best  alternative. 

There  are  other  cases  where  multiple  alternatives  need  to  be  selected  within  budget 
constraints.  This  is  where  the  power  of  the  PET  serves  to  decrease  evaluation  costs  and 
increase  total  value  received.  The  next  example  illustrates  this  case. 


NAVSEA  CORPORATE  PRODUCTIVITY  FUND  ANALYSIS 
USING  THE  PROPOSAL  EVALUATION  TOOL 

The  NAVSEA  established  a  corporate  productivity  fund  to  facilitate  cost  saving 
initiatives  within  the  organization.  Each  year,  NAVSEA  received  hundreds  of 
initiatives  to  evaluate.  Proposals  were  submitted  and  evaluated  in  terms  of  technical 
benefits  and  cost  savings,  and  the  best  ones  were  chosen  and  funded.  The  NAVSEA 
used  commercially  available  tools  to  do  the  evaluations  which  proved  to  be  a  time- 
consuming  effort.  The  fund  was  eventually  disestablished  and  responsibility  for  cost- 
savings  initiatives  was  allocated  to  lower  levels  in  the  organization. 

The  PET  can  be  effectively  used  at  any  level  of  the  organization.  It  facilitates 
standardized  documentation  of  proposals  and  automates  the  evaluation  and  selection 
process.  The  example  below  takes  the  evaluation  criteria  used  by  the  corporate 
productivity  fund  and  applies  it  to  some  sample  proposals.  The  details  are  omitted  to 
place  more  focus  on  the  capabilities  of  the  tool. 

Evaluation  criteria  and  weighting  were  developed  and  proposals  were  submitted 
to  be  evaluated.  This  was  followed  by  the  evaluation  and  proposals  being  ranked  by 
relative  value.  The  criteria  were  that  the  cumulative  cost  of  selected  proposals  could 
not  exceed  $2.5  million  and  that  the  relative  value  of  the  proposal  had  to  be  greater 
than  five.  Thus,  only  eight  out  of  the  hundreds  of  proposals  submitted  met  the  criteria 
to  be  funded.  This  feature  uses  linear  programming  to  determine  the  best  set  of  choices. 
This  methodology  is  a  big  time  saver  in  evaluating  and  choosing  proposals.  The  added 
benefit  is  that  the  computerized  optimization  program  finds  the  best  solution  out  of 
many  combinations. 

One  feature  that  the  PET  automates  beyond  what  is  currently  available  elsewhere  is 
its  ability  to  do  multiyear  funding  optimization.  Given  hundreds  of  proposals,  it  can 
develop  spending  plans  across  multiple  years  and  projects.  It  is  able  to  maximize  total 
benefits  while  staying  within  yearly  funding  constraints.  To  do  this  it  requires  schedule 
information  and  funding  limits  for  each  project. 
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CONCLUSION 

There  is  a  need  for  improved  teehnology  management  and  proposal  evaluation 
processes  within  the  DoD.  Training  and  tools  need  to  reach  the  personnel  who  are 
designing  and  procuring  our  high-value  weapon  systems.  It  is  hoped  that  this  article 
will  be  a  starting  point  in  this  endeavor  and  serve  as  a  catalyst  for  future  improvements. 
By  using  technical  expertise  to  link  capabilities  to  proposal  rating  criteria,  the  DoD  can 
better  use  standards-based  questions  in  RTFs  that  are  based  on  detailed  requirements 
documentation.  Use  of  these  tools  will  enable  better  designs,  reduce  labor  costs,  reduce 
time,  and  improve  documentation.  It  will  also  enable  better  defense  of  source  selections 
by  procurement  officials. 


AUTHOR'S  NOTE 

For  more  information  on  the  TIES  method,  contact  Dr.  Dimitri  Mavris  at  the  Georgia 
Institute  of  Technology  or  see  the  following  Web  site:  www.asdl.gatech.edu 


Stephen  J.  Moretto  is  assigned  to  the  Department  of  Defense  at  the 
Industrial  College  of  the  Armed  Forces.  His  prior  assignments  include 
Navy  Engineering  and  Science  Community  Manager,  Director  of 
Analysis  and  Evaluation  in  the  Office  of  Naval  Research,  Information 
Technology  Department  Head  at  Carderock,  Deputy  Director  of 
the  Center  for  Innovation  in  Ship  Design,  and  Aircraft  Carrier  Cost 
Engineering  Team  Leader  in  NAVSEA. 

(E-mail  address:  Morettos@ndu.edu) 
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